Automated machine vision-based defect detection

ABSTRACT

Provided are various mechanisms and processes for automatic computer vision-based defect detection using a neural network. A system is configured for receiving historical datasets that include training images corresponding to one or more known defects. Each training image is converted into a corresponding matrix representation for training the neural network to adjust weighted parameters based on the known defects. Once sufficiently trained, a test image of an object that is not part of the historical dataset is obtained. Portions of the test image are extracted as input patches for input into the neural network as respective matrix representations. A probability score indicating the likelihood that the input patch includes a defect is automatically generated for each input patch using the weighted parameters. An overall defect score for the test image is then generated based on the probability scores to indicate the condition of the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. patent application Ser. No.17/110,131, entitled “AUTOMATED MACHINE VISION-BASED DEFECT DETECTION”,filed Dec. 2, 2020 by Rajen Bhatt et al., which application claims thebenefit of U.S. Provisional Application No. 62/950,440, entitled“AUTOMATED MACHINE VISION-BASED DEFECT DETECTION”, filed on Dec. 19,2019. These applications are incorporated by reference herein in theirentirety for all purposes.

TECHNICAL FIELD

The present disclosure relates generally to inspection of manufacturedparts, and more specifically to automated machine vision-based detectionof defects.

BACKGROUND

Identifying defects is an important component in many manufacturingprocesses. Quality checks in existing systems involve visualconfirmation to ensure the parts are in the correct locations, have theright shape or color or texture, and are free from any blemishes such asscratches, pinholes, and foreign particles. However, human visualinspection may not be reliable due to limitations of human vision andhuman error. Additionally, the volume of inspections, product variety,and the possibility that defects may occur anywhere on the product andcould be of any size may prove to be a heavy burden for inspectors.Therefore, there is a need for efficient systems and methods to replacehuman visual inspection of machine manufactured parts.

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding of certain embodiments of thedisclosure. This summary is not an extensive overview of the disclosureand it does not identify key/critical elements of the disclosure ordelineate the scope of the disclosure. Its sole purpose is to presentsome concepts disclosed herein in a simplified form as a prelude to themore detailed description that is presented later.

In general, certain embodiments of the present disclosure describesystems and methods for automated machine vision-based defect detection.The method comprises operating in a training mode and in an inferencemode. The method comprises training a neural network to detect defects.Training the neural network includes receiving a plurality of historicaldatasets including a plurality of training images corresponding to oneor more known defects, converting each training image into acorresponding matrix representation, and inputting each correspondingmatrix representation into the neural network to adjust weightedparameters based on the one or more known defects. The weightedparameters correspond to dimensions of the matrix representations. Themethod further comprises obtaining a test image of an object. The testimage is not part of the historical dataset.

The method further comprises extracting portions of the test image as aplurality of input patches for input into the neural network, with eachinput patch corresponding to an extracted portion of the test image. Themethod further comprises inputting each input patch into the neuralnetwork as a respective matrix representation to automatically generatea probability score for each input patch using the weighted parameters.The probability score for each input patch indicates the probabilitythat the input patch includes a predicted defect, and a defect score forthe test image is generated based on the probability scores for eachinput patch. The defect score indicates a condition of the object.

The input patches may include a uniform height and a uniform width. Theinput patches may include overlapping portions of the test image. Theinput patches may be aligned such that each input patch is immediatelyadjacent to one or more other input patches of the plurality of inputpatches.

The neural network may comprise one or more of the following: aconvolution layer, a max pooling layer, a flattening layer, and a fullyconnected layer. The neural network may be trained to accurately outputprobability scores for input patches with unknown defects using theweighted parameters. The method may further comprise generating a heatmap of the input patches based on the probability scores. Prior topassing the test image into the neural network, the test image may bepre-processed to remove a background and represent the image in only aluma component of YCbCr format.

Other implementations of this disclosure include corresponding devices,systems, and computer programs configured to perform the describedmethods. These other implementations may each optionally include one ormore of the following features. For instance, provided is a serversystem comprising an interface configured to receive a plurality ofhistorical data sets including a plurality of images corresponding toone or more levels of known defects, and a test image of an object. Thetest image is not part of the historical dataset. The system furthercomprises memory configured to store the historical datasets and thetest image.

The system further comprises a processor associated with a neuralnetwork. The configured for training a neural network to detect defects.Training the neural network includes converting each training image intoa corresponding matrix representation, and inputting each correspondingmatrix representation into the neural network to adjust weightedparameters based on the one or more known defects. The weightedparameters correspond to dimensions of the matrix representations.

The processor is further configured for extracting portions of the testimage as a plurality of input patches for input into the neural network,with each input patch corresponding to an extracted portion of the testimage. The processor is further configured for inputting each inputpatch into the neural network as a respective matrix representation toautomatically generate a probability score for each input patch usingthe weighted parameters. The probability score for each input patchindicates the probability that the input patch includes a predicteddefect, and a defect score for the test image is generated based on theprobability scores for each input patch. The defect score indicates acondition of the object.

Also provided are one or more non-transitory computer readable mediahaving one or more programs stored thereon for execution by a computerto perform the described methods and systems. These and otherembodiments are described further below with reference to the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings, whichillustrate particular embodiments of the present disclosure.

FIG. 1A illustrates a diagram of an example network architecture forimplementing various systems and methods of the present disclosure, inaccordance with one or more embodiments.

FIG. 1B illustrates an example imaging and processing system forautomated inspection of manufactured parts, in accordance with one ormore embodiments.

FIG. 2 illustrates a process flow chart for automated machinevision-based defect detection, in accordance with one or moreembodiments.

FIGS. 3A and 3B illustrate images captured for inspection of parts, inaccordance with one or more embodiments.

FIGS. 4A and 4B illustrate example output images resulting fromautomated inspection, in accordance with one or more embodiments.

FIG. 5 illustrates an example user interface displaying processed andinspected images, in accordance with one or more embodiments.

FIG. 6 illustrates an example neural network architecture implemented toautomatically detect defects, in accordance with one or moreembodiments.

FIGS. 7A, 7B, and 7C illustrate an example method for automated machinevision-based defect detection, in accordance with one or moreembodiments.

FIG. 8 illustrates a particular example of a computer system that can beused with various embodiments of the present disclosure.

DESCRIPTION OF PARTICULAR EMBODIMENTS

Reference will now be made in detail to some specific examples of thepresent disclosure. Examples of these specific embodiments areillustrated in the accompanying drawings. While the present disclosureis described in conjunction with these specific embodiments, it will beunderstood that it is not intended to limit the present disclosure tothe described embodiments. On the contrary, it is intended to coveralternatives, modifications, and equivalents as may be included withinthe spirit and scope of the present disclosure as defined by theappended claims.

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present disclosure.Particular example embodiments of the present disclosure may beimplemented without some or all of these specific details. In otherinstances, well known process operations have not been described indetail in order not to unnecessarily obscure the present disclosure.

Various techniques and mechanisms of the present disclosure willsometimes be described in singular form for clarity. However, it shouldbe noted that some embodiments include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. Furthermore, the techniques and mechanisms of the presentdisclosure will sometimes describe a connection between two entities. Itshould be noted that a connection between two entities does notnecessarily mean a direct, unimpeded connection, as a variety of otherentities may reside between the two entities. Consequently, a connectiondoes not necessarily mean a direct, unimpeded connection unlessotherwise noted.

Overview

The general purpose of the present disclosure, which will be describedsubsequently in greater detail, is to provide a system and method forautomated computer vision solutions to replace human visual inspectionof machine-manufactured parts. Human visual inspection of partsgenerally takes about 30 seconds to 1 minute and always include a chancefor human error. The described systems and associated methods maysignificantly reduce inspection time and provide increased accuracy indetermining defective parts.

The described systems include light sources and high resolution imagingdevices for capturing high resolution images of the machine-manufacturedparts. The image is processed to remove background and other noise,align the image, and implement other image enhancements. Finally, theimage is segmented into input patches for input into a computervision-based model, or neural network, for analysis.

The neural network may comprise various computational layers, includingat least one series of convolution and max pooling layers, a flatteninglayer, and one or more fully connected layers. The neural network istrained to accurately output a probability score for each input patchcorresponding to the likelihood that the input patch includes an imageof a defect. Such defects may be scratches, indents, or any othercondition that does not meet quality standards for the part.

An overall defect score may then be generated for the entire image ofthe part based on the probability scores for each input patch. If theoverall defect score is below a predetermined threshold, the partcorresponding to the image may be classified as satisfactory. However,if the overall defect score is greater than a predetermined threshold,the part may be classified as defective. Defective parts may be removedfrom the assembly line. In some embodiments, defective parts may bediscarded or repaired to meet quality standards.

Various output images may be generated and displayed at a userinterface. For example, at heat map may be generated to indicate theprobability scores for each input patch. As another example, outlines ofthe areas with detected defects may be overlaid onto the captured imageto locate the defects.

Such imaging techniques may provide more accurate and precise analysisof parts compared to human visual inspection. By pre-processing images,surface features may be enhanced for visualization. The describedtechniques may also provide faster review of more parts in a givenamount of time without reducing quality of the detection.

The defect detection process may be implemented at various points in theassembly line in order to reduce production costs or identifymalfunctioning components along the assembly line. For example,defective parts may be identified and discarded by the described systemsbefore additional machining or processing can be performed on such partsin order to avoid unnecessary production costs. As another example, thedescribed techniques may pinpoint and identify issues with processing ormanufacturing components if a high percentage of similar defects arefound after particular points in the assembly line.

Other objectives and advantages of the present apparatus, systems, andmethods will become obvious to the reader and it is intended that theseobjectives and advantages are within the scope of the present invention.

To the accomplishment of the above and related objectives, the disclosedapparatus, systems and methods may be embodied in the form illustratedin the accompanying drawings, attention being called to the fact,however, that the drawings are illustrative only, and that changes maybe made in the specific construction illustrated.

Detailed Embodiments

Turning now descriptively to the drawings, in which similar referencecharacters denote similar elements throughout the several views, theattached figures illustrate systems and methods for automated machinevision-based defect detection.

According to various embodiments of the present disclosure, FIG. 1Aillustrates a diagram of an example network architecture 100 forimplementing various systems and methods of the present disclosure, inaccordance with one or more embodiments. The network architecture 100includes a number of client devices (or “user devices”) 102-108communicably connected to one or more server systems 112 and 114 by anetwork 110. In some implementations, the network 110 may be a publiccommunication network (e.g., the Internet, cellular data network, dialup modems over a telephone network) or a private communications network(e.g., private LAN, leased lines).

In some embodiments, server systems 112 and 114 include one or moreprocessors and memory. The processors of server systems 112 and 114execute computer instructions (e.g., network computer program code)stored in the memory to receive and process data received from thevarious client devices. In some embodiments, server system 112 is acontent server configured to receive and store historical data sets,parameters, and other training information for a neural network. In someembodiments server system 114 is a dispatch server configured totransmit and/or route network data packets including network messages.In some embodiments, content server 110 and dispatch server 114 areconfigured as a single server system that is configured to perform theoperations of both servers.

In some embodiments, the network architecture 100 may further include adatabase 116 communicably connected to client devices 102-108 and serversystems 112 and 114 via network 110. In some embodiments, network data,or other information such as computer instructions, historical datasets, parameters, and other training information for a neural networkmay be stored in and/or retrieved from database 116.

Users of the client devices 102-108 access the server system 112 toparticipate in a network data exchange service. For example, the clientdevices 102-108 can execute web browser applications that can be used toaccess the network data exchange service. In another example, the clientdevices 102-108 can execute software applications that are specific tothe network (e.g., networking data exchange “apps” running on devices,such as computers or smartphones).

Users interacting with the client devices 102-108 can participate in thenetwork data exchange service provided by the server system 112 bydistributing and retrieving digital content, such as text comments(e.g., updates, announcements, replies), digital images, videos, onlineorders, payment information, activity updates, location information,computer code and software, or other appropriate electronic information.In some embodiments, network architecture 100 may be a distributed, openinformation technology (IT) architecture configured for edge computing.

In some implementations, the client devices 102-108 can be computingdevices such as laptop or desktop computers, smartphones, personaldigital assistants, portable media players, tablet computers, cameras,or other appropriate computing devices that can be used to communicatethrough the network. In some implementations, the server system 112 or114 can include one or more computing devices such as a computer server.In some implementations, the server system 112 or 114 can represent morethan one computing device working together to perform the actions of aserver computer (e.g., cloud computing). In some implementations, thenetwork 110 can be a public communication network (e.g., the Internet,cellular data network, dial up modems over a telephone network) or aprivate communications network (e.g., private LAN, leased lines).

In various embodiments, the client devices and/or servers may beimplemented as an imaging and image processing system. FIG. 1Billustrates such an example imaging and processing system 150 forautomated inspection of manufactured parts, in accordance with one ormore embodiments. In various embodiments, system 150 includes platform152 with one or more light sources 160 positioned around the platform.Object 310 may be placed upon the surface of the platform. In someembodiments, the platform may be configured to secure object 310 in adesired position or orientation. Object-securing mechanisms may includefasteners, clamps, vacuum-based holders, etc. Although four lightsources 160 are shown positioned at the corners of platform 152, variousembodiments may include more or fewer light sources positioned invarious other locations to provide the desired illumination of object310. In some embodiments, the positions of light sources 160 may beconfigured to be changed to desired positions during operation toprovide the desired lighting upon the object. Any suitable movementmechanism (such as motors, etc.) for positioning the light sources maybe implemented.

System 150 may further include camera 154. In various embodiments,camera 154 is a high resolution camera configured to take highresolution still images of objects on the platform. The capture imagesmay then be transmitted to processing device 156 which may apply imageprocessing algorithms and implement computer vision-based modelsdescribed herein to automatically detect defects on the object. As usedherein, computer vision-based models may include neural networks.

In various embodiments, processing device 156 may be an edge computingdevice configured to locally process the images captured from camera 154using computer vision models described herein. In some embodiments,processing device 156 is an embedded device in a client device (such ascamera 154) that performs the image processing described herein. In someembodiments, the embedded device is a microcontroller unit (MCU) orother embedded processor or chip. In some embodiments, client devices102-108 may function as processing device 156 to perform the imageprocessing. In some embodiments, processing device 156 may be servers112 and/or 114 that are implemented as local computers or servers on aprivate LAN to process the captured images. In some embodiments servers112 and/or 114 may be implemented as a centralized data center providingupdates and parameters for a neural network implemented by theprocessing device. Such edge computing configurations may allow forefficient data processing in that large amounts of data can be processednear the source, reducing Internet bandwidth usage. This both reducescosts and ensures that applications can be used effectively in remotelocations. In addition, the ability to process data without ever puttingit into a public cloud adds a useful layer of security for sensitivedata.

FIG. 2 illustrates a process flow chart for automated machinevision-based defect detection, in accordance with one or moreembodiments. At operation 202, an object is obtained for imaging. Inparticular embodiments, object 310 is a machine manufactured part. Forexample, object 310 may be a garnish for an automobile, such as moldingtrim.

At operation 204, the object is positioned into a desired orientation.For example, the part may be positioned and secured onto platform 152.In some embodiments, such parts may be machined by various automatedprocesses and directly placed on the platform. In some embodiments, theplatform may be integrated into the assembly line such that parts may beinspected at various times in the manufacturing process. For example,automotive garnish parts may have a scratch (or multiple scratches)which does not pass predetermined quality standards. Such defectiveparts may then be discarded or further processed to address the defects.Parts which do not indicate any scratches or defects are acceptable andcan pass the quality standard for further processing.

Once positioned in the desired orientation on the platform, the objectis exposed to sufficient lighting and still images are captured bycamera 154, which may obtain high resolution images of the object atoperation 206. For example, a capture image may include about 8megabytes, or a resolution above about 1800×1200 pixels, or an effectiveresolution above about 300 pixels per inch. With reference to FIG. 3A,shown is a high resolution image 300 of object 310. As shown, image 300includes object 310 along with background 312 and shadow 314.

At operation 208, the high resolution image is pre-processed to preparethe image for input into the described neural network. In someembodiments, the image may be pre-processed to sharpen the image toenhance fine details of the imaged object. In some embodiments, otherpre-processing stages may include automatic alignment of the object,background removal, color removal, contrast enhancement, and other imagequality enhancements.

With reference to FIG. 3B, shown is an example of a pre-processed orenhanced image 320 of object 310, in accordance with one or moreembodiments. Image 320 has been pre-processed to remove the backgroundand increase contrast. Furthermore, image 320 is represented in only asingle channel, specifically the Y component of YCbCr format. This colorremoval may enhance any surface defects that are present.

At operation 210, portions of the enhanced image are extracted as inputpatches. In various embodiments, the system extracts uniform portions ofthe pre-processed image that include the same pixel dimensions. Forexample, the input patches may each be 64 by 64 pixels in dimension.However, other sizes for the input patches may be determined by theconfiguration of the system. The input patches may be extracted as twodimensional segments of the image corresponding to the Y component.However, in some embodiments, the patches may include a third dimensionif some color component or channel is included in the pre-processedimage.

Several examples of input patches are shown in FIG. 3B. In someembodiments, the input patches include overlapping portions of theenhanced image. For example, patches 322, 324, and 326 includeoverlapping portions of image 320. Input patch 322 is shown outlinedwith a different line pattern for illustrative purposes. In suchembodiments, each patch may overlap with neighboring patches by the samepredetermined amount. By inputting overlapping images, portions of theobject may be analyzed by the model more than once, thereby increasingthe accuracy of the final defect score. However, by overlapping inputpatches, more input patches will be required to input the entireenhanced image through the neural network, thereby requiring additionalprocessing time and resources.

As another example, input patches may exactly border adjacent patches.This allows the entire image to be fed into the neural network whileminimizing the amount of necessary patches and therefore reduce therequired processing time and resources. For example, patches 330, 331,and 332 are immediately adjacent to each other such that the pixels atthe edge of adjacent patches are positioned immediately next to eachother in image 320.

In yet other embodiments, extracted patches may be separated a number ofpixels thereby further decreasing processing requirements, but at theexpense of some accuracy due to the fact that not all portions of theobject or enhanced image will be input into the neural network. Forexample, patches 340, 341, and 342 are separated by from each other by aset distance.

At operation 212, the input patches are passed into the describedcomputer vision-based model, or neural network. In various embodiments,the input patches are input as pixel matrices. For example, the systemmay convert each patch into a matrix with dimensions equal to the pixeldimensions of the patch. Each pixel may be presented by one matrixelement and assigned a value based on the shade of the pixel. Forexample, each matrix element may correspond to an integer from the set{0, 1, 2, . . . 255} where 0 corresponds to black and 255 corresponds towhite. In described particular example, each input patch is 64×64pixels. Such input patch would result in a matrix that is 64×64×1.

The input patches may then be fed into the neural network sequentiallyor in parallel based on the system architecture. As previouslydescribed, the system architecture may comprise a processing deviceimplemented as an embedded target designed for specific controlfunctions within a larger system, often with real-time computingconstraints. Such embedded target may be embedded as part of a completedevice often including hardware and mechanical parts. For example, theembedded target may be an embedded microcontroller unit (MCU) orembedded processor of the camera, which implements the neural network.In various embodiments, the neural network is stored in flash memory orother storage corresponding to the embedded target, or on otheraccessible memory of the camera. In other embodiments, the processingdevice may be implemented as a local or cloud-based server. In edgecomputing configurations, large amounts of data may be processed nearthe source, reducing Internet bandwidth usage, allowing for images to beinput in parallel. However, where the processing device is implementedas a centralized cloud-based server, additional processing time andpower may be required to transmit the images to the server forprocessing, requiring images to be input sequentially.

In some embodiments, only input patches containing portions of theobject are input into the neural network. Various object recognitiontechniques may be implemented to identify input patches that do notinclude any part of the object, such as patches 340 and 341. This mayreduce the overall processing requirements by preventing analysis ofinput patches that do not include any portion of the imaged object.

At operation 214 a probability score is output by the computervision-based model for each input patch that is passed into the model.For example, a probability score between 0 and 1 may be determined foreach input patch, which indicates the likelihood that the image in theinput patch includes a defect. As such, a score of 0 would indicate nodefect detected and a score of 1 would indicate a positive detection ofa defect. In other words, a probability score of 1 means that the modelis 100% confident of a defect shown in the input patch, whereas anoutput probability score of 0.87 means that the model is 87% confidentof the presence of a defect.

In various embodiments, the model is trained to determine a probabilityscore based on several factors. For example, the size and deepness of ascratch on the part, as represented by the image in the input patch, mayaffect the probability score. In various embodiments, the probabilityscore may be visualized for review by a user. With reference to FIG. 4A,shown is an example heat map 410 of the input patches reflecting thedetermined probability scores. The axes of heat map 410 indicate thatthe image is approximately 3840×880 pixels.

The scale 412 included in FIG. 4A indicates that the probability scoresare represented with shading from black (indicating a score of 0.00) towhite (indicating a score of 1.00). In various embodiments, an area ofimage 410 corresponding to the input patch is shaded based on thepredicted presence of a defect within that patch. Thus, the shadedpatches indicate locations and severity of estimated defects on thepart. The shaded patches in FIG. 4A are shown to overlap, indicating theoverlapping portions of the extracted input patches

At operation 216, an overall defect score is determined for the object.The overall defect score may be determined based on the probabilityscores for each of the input patches. In some embodiments, the overalldefect score is the maximum value of the accumulated probability scores.For example, p(s1) identifies the probability of a defect for a firstpatch, p(s2) identifies the probability of a defect for a second patch,and so on up to p(sN) for the Nth patch. The overall defect score may bedetermined as max{p(s1), p(s2), . . . , p(sN)}. However, in someembodiments, the overall defect score may be determined based on othermethods. For example, the overall defect score may be determined basedon an average of the accumulated probability scores.

In some embodiments, a part is determined to be unacceptably defectiveif the overall defect score is above a predetermined threshold. Forexample, a part with an overall defect score greater than 0.90 may bedeemed to be unacceptably defective. Referring back to the example ofFIG. 4A, the maximum of the probability scores is 0.93, thus the overalldefect score is 0.93.

With reference to FIG. 4B, shown is an example image 420 produced by thedescribed systems, in accordance with one or more embodiments. Image 420depicts a part with outlined areas corresponding to defects detected bythe model. In some embodiments, the outlined areas may correspond to theportions of the image included in the input patches with a probabilityscore above a predetermined threshold. For example, the outlined areasmay correspond to input patches with assigned probability scores greaterthan 0.90.

One or more of the various images previously described may be displayedat a user interface. With reference to FIG. 5 , shown is an example userinterface 500 displaying processed and inspected images, in accordancewith one or more embodiments. Images 300, 320, and 420 are displayed atuser interface 500. This may allow a user of the system to visuallyreview the analysis performed by the model. In some embodiments, aquality control status 510 may be displayed indicating the acceptabilityof the part. In some embodiments, the overall defect score may also beshown.

At operation 218, the object may be further processed based on thedetermined defect score. In some embodiments, the described methods ofdefect detection may be performed after the machining to analyze thefinal output part. Parts found to be acceptable (such as those withdefect scores at or below 0.90) may be transferred for packaging orshipment. However, the described models may be implemented at variouspoints in the assembly line, and at multiple points in the assemblyline.

In some embodiments, the part may be repaired to correct the defects.For example, the part may be automatically transferred to another areaof the assembly line to correct the defects found. As another example, adefective part may be disposed of. In some embodiments, defective partsmay be re-machined or recycled to form new parts. Implementing thecomputer vision-based model at various points can identify defectiveparts before further manufacturing is performed on the defective parts,saving resources, materials, and costs. The quick automatic defectdetection provided by the model may also be used at various pointsduring the manufacturing process in order to manage the performance ofparticular components of the assembly line and pinpoint potentialissues. For example, if a high percentage of parts are found to bedefective after point B in an assembly line, but the same parts areacceptable after a previous point A, then it may suggest an issue withthe machining tools beginning at point B.

The computer vision-based model may be a neural network with variouscomputational layers. With reference to FIG. 6 , shown is an exampleneural network architecture 600 implemented to automatically detectdefects, in accordance with one or more embodiments. As shown, neuralnetwork 600 includes convolution layer 612, max pooling layer 614,flattening layer 616, fully connected layer 618, and fully connectedlayer 620.

An input patch 602 may be input into the convolution layer 612. Invarious embodiments, the input patch 602 may be an extracted portion ofan image, such as input patch 330. In some embodiments, input patch 602may be a portion of an image with an unknown defect status. In someembodiments, the input patch 602 may be a training image with a knowncorresponding defect. For example, a training image may include acorresponding probability score of 0 (indicating no defects) or 1(indicating a defect).

In various embodiments, convolution layer 612 applies a filter, K, ofparticular dimensions to the pixel matrix of the input patch. Forexample, the filter may include the dimensions of 3×3×1. In someembodiments, the filter is applied with a stride length of 8. Theconvolution operation extracts high-level features from the input patch.The convolution layer outputs a convolved matrix. The convolution layermay apply same padding or valid padding to the matrix to output theconvolved matrix.

The convolved matrix output is then fed into the max pooling layer 614.In various embodiments, the max pooling layer performs max pooling ofthe convolved matrix by returning the maximum value from the portion ofthe convolved matrix covered by the max pooling kernel. For example, thepool size may be 2×2×1. In some embodiments, the neural network mayapply an average pooling function instead of max pooling, which returnsthe average of all the values from the portion of the convolved matrixcovered by the max pooling kernel. In an example, the output of the maxpooling layer may be a matrix of 64 units (a 64×64 matrix).

As such, the pooling layer may reduce the spatial size of the convolvedfeature in order to decrease the computational power required to processthe data through dimensionality reduction, as well as to extractdominant features for maintaining the process of training the model. Insome embodiments, the neural network may include a series of consecutiveconvolution and max pooling layers. For example, neural network 600 mayinclude three consecutive convolution-pooling pairs 615 in which theoutput of the max pooling layer is fed as input into the convolutionlayer of a subsequent convolution-pooling pair. The convolution and maxpooling layers may implement a truncated normal distribution forinitialization and a rectified activation function. As such, eachconvolution-pooling pair 615 may take a matrix of 64 units as input andoutput a matrix of 64 units.

The neural network may include any number of consecutiveconvolution-pooling pairs based on available processing resources anddesired performance. Implementation of three consecutiveconvolution-pooling pairs may minimize the latency of the imageprocessing while maintaining a desired level of accuracy in prediction.For example, using three convolution-pooling pairs in the neural networkmay allow each input patch of a test image to be fully analyzed todetermine a defect score for the object within approximately 5 seconds.The use of a stride length of 8 may further optimize the accuracy andlatency of the image processing (or runtime) based on the number ofplacements of the filter used on each input patch. As such, theinference process may be highly optimized to run from mobile devices orconstrained embedded devices.

The output of the final max pooling layer is then fed into flatteninglayer 616 to flatten the output into a column vector. The column vectoroutput is then fed into fully connected layers 618 and 620. In variousembodiments, the fully connected layers may be a multi-layer perceptron(a feed-forward neural network). In some embodiments, the first fullyconnected layer 618 implements a rectified linear unit (ReLU) as anactivation function. In an example embodiment, the first fully connectedlayer 618 may comprise 128 neurons. However, a greater or a fewer numberof neurons may be implemented in different embodiments. In someembodiments, the second fully connected layer 620 implements a sigmoidactivation function. In some embodiments, the fully connected layers mayimplement a truncated normal distribution for initialization.

During a training mode, neural network 600 may be configured to produceprobabilities that a particular input patch includes a defect. Invarious embodiments, output 630 may be set to a probability score of 1if the training image includes a known defect, or to a probability scoreof 0 if the training image does not include any defect. With the knownprobability score, the weights (or parameters) in the fully connectedlayers may be updated using backpropagation. For example, the parametersmay be updated via a stochastic gradient descent algorithm with an Adamoptimization algorithm. In some embodiments, this may be achieved byconverting activation values of output layer neurons to probabilitiesusing a softmax function.

In some embodiments, the training of the neural network may be performedat a centralized server system in a global or cloud network. In someembodiments, the training data, such as weights, parameters, andtraining images may be stored at the centralized server system. Theupdated weights may then be transmitted from the centralized serversystem to a local edge computing device for more efficient imageprocessing. As previously described, the local edge computing device maybe an embedded target, such as an MCU or an embedded processor, of theclient device, such as camera 154. In some embodiments, the parametersof the neural network may be periodically updated at the centralizedserver based on new training data. However, in some embodiments,training of the neural network may be performed at the local edgecomputing device.

In some embodiments, the neural network is sufficiently trained once apredetermined number of training images have been input into the model.In some embodiments, the neural network is sufficiently trained once itis able to generate predictions with a desired accuracy rate.

Once fully trained, the neural network may then operate in an inferencemode to take an input patch with unknown defect characteristics as input602. The neural network then passes the input through the describedlayers and generates an output 630 for the input patch between 0 and 1based on the updated weights to indicate the probability that the inputpatch includes a defect.

With reference to FIGS. 7A, 7B, and 7C, shown is an example method 700for training and operating a neural network to computer vision-baseddefect detection. The neural network may be neural network 600 and maycomprise one or more computational layers 702. As previously discussed,may comprise one or more of the following layers: a convolution layer, amax pooling layer, a flattening layer, and a fully connected layer. FIG.7B illustrates an example of operations of the neural network in atraining mode 710, and FIG. 7C illustrates an example of operations ofthe neural network in an inference mode 730, in accordance with one ormore embodiments.

In the training mode, the neural network is trained to detect defectsusing datasets of training images. When operating in the training mode710, a plurality of historical datasets is received at operation 711.The historical datasets may include a plurality of training images 717corresponding to one or more known defects. In some embodiments, thetraining images may represent or correspond to input patches extractedfrom images of one or more objects. In some embodiments, the trainingimages may include corresponding values indicating whether the trainingimage includes a defect on the corresponding portion of the object. Forexample, the training image may be associated with a probability scoreof 1 if the training image shows a relevant defect, or a probabilityscore of 0 if the training image does not show a relevant defect. Suchvalues may be stored in the image file of the training images, such asin metadata as an example.

At operation 713, each training image is converted into a correspondingmatrix representation. As previously described, the matrixrepresentation may correspond to the pixel dimensions of the trainingimage. For example, the training image may be 64×64 pixels andrepresented in only one color channel (luma). As such, the dimension ofthe corresponding matrix may be 64×64×1.

At operation 715, each corresponding matrix representation is input intothe neural network to adjust weighted parameters 719 in the variouslayers of the neural network based on the one or more known defects. Insome embodiments, the weighted parameters 719 may correspond todimensions of the matrix representations. The known probability scoresmay be input into the neural network along with the matrixrepresentation to generate and update parameters in the fully connectedlayers of the neural network. As such, the neural network may be trained(721) to accurately output probability scores for input patches withunknown defects using the weighted parameters 719.

In some embodiments, the predictive merchant association model may bedetermined to be sufficiently trained once a desired error rate isachieved. For example, a desired error rate may be 0.00001% (or anaccuracy rate of 99.9999%). In other embodiments, the model may bedetermined to be sufficiently trained after a set number of epochs oriterations, such as after a predetermined number of training images havebeen input into the model. For example, the model may be sufficientlytrained when 1000 training images have been input into the neuralnetwork along with known probability scores. Once sufficiently trained,the neural network may be implemented to detect defects in new images inthe inference mode 730.

When operating in the inference mode 730, a test image 743 of an object,such as object 310, is obtained at operation 731. The test image 743 isnot part of the historical dataset and may include a part with unknownpossible defects. For example test image 743 may be obtained of a partduring the manufacturing process at one of various different points onthe assembly line. The test image may then be pre-processed at operation733 before input into the neural network for analysis. In someembodiments, the test image is pre-processed to remove the backgroundfrom the image of the part. In some embodiments, the test image ispre-processed to represent the image in only a luma component of YCbCrformat. Various other image pre-processing techniques may be implementedon the test image, as previously discussed with reference to operation208.

At operation 735, portions of the test image are extracted as aplurality of input patches 745 for input into the neural network. Forexample, the input patches may be any one of the input patches describedwith reference to FIG. 3B. Each input patch 745 may correspond to anextracted portion of the test image. The pixel dimensions of the inputpatches may be the same as the training images.

At operation 737, each input patch is input into the neural network toautomatically generate a probability score 749 for each input patch 745using the weighted parameters 719. Each input patch 745 may be inputinto the neural network as a respective matrix representation 747,similar to the training images 717. As described, the input patches maybe input into the neural network in series or in parallel. Theprobability score 749 for each input patch indicates the probabilitythat the input patch includes a predicted defect.

Once probability scores have been determined for input patchescorresponding to every portion of the test image, a defect score 751 isgenerated for the test image based on the probability scores for eachinput patch at operation 739. The defect score 751 may indicate acondition of the object. In some embodiments, the defect score may bethe maximum of the determined probability scores 749. For example, adefect score above a predetermined threshold may be determined to beunfit for sale or use. As another example, the defect score may be anaverage of the probability scores.

Parts with defect scores above a predetermined threshold may be disposedof so that they are not used. In some embodiments, defective parts maybe further processed to repair or remove the identified defects. Theanalysis of the images may be visualized for review by a user of thesystem. For example, a heat map of the input patches, such as heat map410, may be generated based on the probability scores at operation 741.Other output images may be generated such as image 420. These outputimages may be displayed at a user interface, such as interface 500, suchthat a user of the system may view the detected defects. This may allowa user to locate defects in order to remove or repair them.

In some embodiments, the predicted defects within the test images orcorresponding input patches may be confirmed at operation 743 and usedto further train and fine tune the neural network. For example, theprobability scores may be confirmed by a user at a user interfacedisplaying the input patch image and corresponding probability score.The user may then confirm whether the image, or particular patches,shows a defect. If the user confirms that a defect is present, theassociated probability score for the input patch may be set at 1. If theuser confirms that no defect is present, the associated probabilityscore for the input patch may be changed to 0.

The input patches selected for confirmation at operation 743 may berandomly selected from one or more different test images obtained duringthe inference mode. However, in some embodiments, input patches with aprobability score within a predetermined range may be selected forconfirmation. For example, input patches receiving a probability scorebetween 0.4 and 0.6 may be selected for confirmation. These images maycorrespond to instances where the neural network is unable to identify adefect with sufficient certainty.

Once input patches have been confirmed, they may be input into theneural network during the training mode to refine the weightedparameters of the neural network. For example, the method may return tooperation 713 or 715 to convert and input a confirmed input patch as atraining image with the confirmed probability score. In someembodiments, the processed input patches are transmitted back to retrainthe neural network in regular batch sizes, which may include apredetermined number of processed input patches, such as 100 inputpatches. For example, a batch of confirmed input patches may comprise ahistorical dataset that is received at operation 711. This improves thenetwork performance over the time and as it sees more examples.

With reference to FIG. 8 , shown is a particular example of a computersystem that can be used to implement particular examples of the presentdisclosure. For instance, the computer system 800 may represent a clientdevice, server, or other edge computing device according to variousembodiments described above. According to particular exampleembodiments, a system 800 suitable for implementing particularembodiments of the present disclosure includes a processor 801, a memory803, an accelerator 805, an interface 811, and a bus 815 (e.g., a PCIbus or other interconnection fabric). When acting under the control ofappropriate software or firmware, the processor 801 is responsible fortraining and implementing described computer models and neural networks.The processor may also be responsible for controlling operationalfunctions of a camera, and transmitting data over a network betweenclient devices and a server system. Various specially configured devicescan also be used in place of a processor 801 or in addition to processor801. The complete implementation can also be done in custom hardware.

The interface 811 may include separate input and output interfaces, ormay be a unified interface supporting both operations. When acting underthe control of appropriate software or firmware, the processor 801 isresponsible for such tasks such as implementation of a neural network orcomputer vision-based model. Various specially configured devices canalso be used in place of a processor 801 or in addition to processor801. The complete implementation can also be done in custom hardware.The interface 811 is typically configured to send and receive datapackets or data segments over a network. Particular examples ofinterfaces the device supports include Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces, andthe like. Generally, these interfaces may include ports appropriate forcommunication with the appropriate media. In some cases, they may alsoinclude an independent processor and, in some instances, volatile RAM.The independent processors may control such communications intensivetasks as packet switching, media control and management.

In addition, various very high-speed interfaces may be provided such asfast Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces,HSSI interfaces, POS interfaces, FDDI interfaces and the like.Generally, these interfaces may include ports appropriate forcommunication with the appropriate media. In some cases, they may alsoinclude an independent processor and, in some instances, volatile RAM.The independent processors may control such communications intensivetasks as packet switching, media control and management.

According to particular example embodiments, the system 800 uses memory803 to store data and program instructions and maintained a local sidecache. The program instructions may control the operation of anoperating system and/or one or more applications, for example. Thememory or memories may also be configured to store received metadata andbatch requested metadata.

In some embodiments, system 800 further comprises a graphics processingunit (GPU) 809. As described above, the GPU 809 may be implemented toprocess each pixel on a separate thread. In some embodiments, system 800further comprises an accelerator 805. In various embodiments,accelerator 805 is a rendering accelerator chip, which may be separatefrom the graphics processing unit. Accelerator 805 may be configured tospeed up the processing for the overall system 800 by processing pixelsin parallel to prevent overloading of the system 800. For example, incertain instances, ultra-high-definition images may be processed, whichinclude many pixels, such as DCI 4K or UHD-1 resolution. In suchinstances, excess pixels may be more than can be processed on a standardGPU processor, such as GPU 809. In some embodiments, accelerator 805 mayonly be utilized when high system loads are anticipated or detected.

In some embodiments, accelerator 805 may be a hardware accelerator in aseparate unit from the CPU, such as processor 801. Accelerator 805 mayenable automatic parallelization capabilities in order to utilizemultiple processors simultaneously in a shared memory multiprocessormachine. The core of accelerator 805 architecture may be a hybrid designemploying fixed-function units where the operations are very welldefined and programmable units where flexibility is needed. In variousembodiments, accelerator 805 may be configured to accommodate higherperformance and extensions in APIs, particularly OpenGL 2 and DX9.

Because such information and program instructions may be employed toimplement the systems/methods described herein, the present disclosurerelates to tangible, machine readable media that include programinstructions, state information, etc. for performing various operationsdescribed herein. Examples of machine-readable media include hard disks,floppy disks, magnetic tape, optical media such as CD-ROM disks andDVDs; magneto-optical media such as optical disks, and hardware devicesthat are specially configured to store and perform program instructions,such as read-only memory devices (ROM) and programmable read-only memorydevices (PROMs). Examples of program instructions include both machinecode, such as produced by a compiler, and files containing higher levelcode that may be executed by the computer using an interpreter.

Although many of the components and processes are described above in thesingular for convenience, it will be appreciated by one of skill in theart that multiple components and repeated processes can also be used topractice the techniques of the present disclosure.

While the present disclosure has been particularly shown and describedwith reference to specific embodiments thereof, it will be understood bythose skilled in the art that changes in the form and details of thedisclosed embodiments may be made without departing from the spirit orscope of the disclosure. It is therefore intended that the disclosure beinterpreted to include all variations and equivalents that fall withinthe true spirit and scope of the present disclosure.

What is claimed is:
 1. A method comprising: obtaining a test image of anobject; extracting portions of the test image as a plurality of patchesfor input into a neural network that was trained using a historicaldataset that does not include the test image, each patch correspondingto an extracted portion of the test image; determining whether eachpatch contains a portion of the object; and inputting each patch that isdetermined to contain a portion of the object, while preventing input ofpatches that are determined to not contain a portion of the object, intothe neural network as a respective matrix representation toautomatically generate a probability score for each patch using theweighted parameters; wherein the probability score for each patch inputinto the neural network indicates the probability that the patchincludes a predicted defect, wherein a defect score for the test imageis generated based on the probability scores for each patch, and whereinthe defect score indicates a condition of the object.
 2. The method ofclaim 1, wherein the neural network is embedded in a camera device. 3.The method of claim 1, wherein the patches are input into the neuralnetwork in parallel.
 4. The method of claim 1, wherein the patchesinclude overlapping portions of the test image.
 5. The method of claim1, wherein the patches are aligned such that each patch is immediatelyadjacent to one or more other patches of the plurality of patches. 6.The method of claim 1, wherein the neural network is configured toaccurately output a probability score for a defect in each patch inputinto the neural network using the weighted parameters.
 7. The method ofclaim 6, further comprising generating a heat map of the patches basedon the probability scores of the patches.
 8. The method of claim 1,further comprising, prior to inputting the patches of the test imageinto the neural network, removing background and other noise from thetest image and removing any color components from the test image to onlyretain a luma component.
 9. A system comprising: an interface configuredto receive a test image of an object; memory configured to store thetest image; and a processor associated with a neural network, whereinthe processor is configured for: extracting portions of the test imageas a plurality of patches for input into a neural network that wastrained using a historical dataset that does not include the test image,each patch corresponding to an extracted portion of the test image;determining whether each patch contains a portion of the object; andinputting each patch that is determined to contain a portion of theobject, while preventing input of patches that are determined to notcontain a portion of the object, into the neural network as a respectivematrix representation to automatically generate a probability score foreach patch using the weighted parameters; wherein the probability scorefor each patch input into the neural network indicates the probabilitythat the patch includes a predicted defect, wherein a defect score forthe test image is generated based on the probability scores for eachpatch, and wherein the defect score indicates a condition of the object.10. The system of claim 9, wherein the neural network is embedded in acamera device.
 11. The system of claim 9, wherein the patches are inputinto the neural network in parallel.
 12. The system of claim 9, whereinthe patches include overlapping portions of the test image.
 13. Thesystem of claim 9, wherein the patches are aligned such that each patchis immediately adjacent to one or more other patches of the plurality ofpatches.
 14. The system of claim 9, wherein the neural network isconfigured to accurately output a probability score for a defect in eachpatch input into the neural network using the weighted parameters. 15.The system of claim 14, wherein the processor is further configured forgenerating a heat map of the patches based on the probability scores ofthe patches.
 16. The system of claim 9, wherein the processor is furtherconfigured for, prior to inputting the patches of the test image intothe neural network, removing background and other noise from the testimage and removing any color components from the test image to onlyretain a luma component.
 17. A non-transitory computer readable mediumstoring one or more programs configured for execution by a computer, theone or more programs comprising instructions for: obtaining a test imageof an object; extracting portions of the test image as a plurality ofpatches for input into a neural network that was trained using ahistorical dataset that does not include the test image, each patchcorresponding to an extracted portion of the test image; determiningwhether each patch contains a portion of the object; and inputting eachpatch that is determined to contain a portion of the object, whilepreventing input of patches that are determined to not contain a portionof the object, into the neural network as a respective matrixrepresentation to automatically generate a probability score for eachpatch using the weighted parameters; wherein the probability score foreach patch input into the neural network indicates the probability thatthe patch includes a predicted defect, wherein a defect score for thetest image is generated based on the probability scores for each patch,and wherein the defect score indicates a condition of the object. 18.The non-transitory computer readable medium of claim 17, wherein theneural network is embedded in a camera device.
 19. The non-transitorycomputer readable medium of claim 17, wherein the patches are input intothe neural network in parallel.
 20. The non-transitory computer-readablemedium of claim 17, wherein the neural network comprises one or more ofthe following: a convolution layer, a max pooling layer, a flatteninglayer, and a fully connected layer.